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LONG-TERM  GOALS 

The  long-term  goal  of  this  project  is  to  develop  robust  global  and  mesoscale  ensemble  analysis  and 
forecast  systems  that  are  able  to  provide  probabilistic  forecast  guidance  in  an  operational  environment. 
Although  most  operational  centers  rely  solely  on  single  deterministic  forecasts  from  numerical  weather 
prediction  models,  recent  advances  in  probabilistic  prediction,  or  ensemble  forecasting,  have  made  this 
a  very  powerful  tool  for  providing  more  complete  input  on  environmental  conditions,  including  a 
measure  of  the  forecast  confidence.  Since  the  atmosphere  is  inherently  chaotic,  the  model  predictions 
of  most,  if  not  all  phenomena  will  always  be  sensitive  to  the  model  initial  conditions  and  physical 
parameterizations.  Mesoscale  models  are  additionally  constrained  by  potential  errors  inherent  in  the 
specification  of  the  lateral  boundary  conditions  (LBC).  Due  to  these  inevitable  uncertainties,  forecasts 
are  most  appropriately  viewed  in  a  probabilistic  sense;  that  is,  forecasts  should  provide  probabilities  of 
the  occurrence  of  specific  high-impact  events,  as  well  as  estimates  of  forecast  skill.  Recent  research  at 
NRL  and  other  research  centers  has  demonstrated  that  probabilistic  information  can  be  obtained  from 
ensemble  of  forecasts  created  from  equally  plausible  initial  states  and  equally-likely  model 
formulations.  However,  such  a  probabilistic  prediction  of  Navy  relevant  high-impact  weather  will 
significantly  benefit  sea  strike  and  sea  shield  functions  only  if  there  are  clearly  identified  user-relevant 
norms.  For  example,  METOC  impacts  on  operations  are  typically  listed  in  a  “stoplight”  format,  often 
with  multiple  weather  criteria  that  must  be  satisfied  (e.g.,  for  the  Predator,  precipitation,  winds,  ceiling, 
visibility,  cloud  cover,  icing,  turbulence,  and  temperature).  Probabilistic  forecasts  allow  for  a  different 
probability  threshold  to  be  set  for  each  criterion  individually  (e.g.  a  90%  probability  for  winds,  but 
only  a  70%  probability  for  visibility),  reflecting  the  relevant  importance  of  each  weather  factor. 
Ensembles  also  allow  for  the  covariance  between  relevant  weather  variables  to  be  taken  into  account. 

In  a  particular  instance,  wind  and  visibility  may  have  a  wide  range  of  values  they  might  take  on  (broad 
probability  distribution  functions),  but  information  in  the  covariance  could  tell  you  that  if  the  wind 
speed  is  high,  then  visibility  will  also  be  high. 

A  prototype  global  Ensemble  Transform  (ET)  ensemble  system  has  been  developed  as  part  of  research 
performed  in  recent  NRL  base  projects  on  global  modeling  and  predictability  (Bishop  and  Toth  1999, 
McLay  et  al.  2007).  In  addition,  a  prototype  ET  mesoscale  ensemble  system,  driven  by  this  global 
ensemble,  has  been  developed  within  the  NRL  6.2  project  "Probabilistic  Prediction  of  High-Impact 
Weather"  that  was  approved  for  FY06-FY09.  The  work  in  this  NRL  base  project  on  mesoscale 
ensembles  has  been  incorporated  into  this  RTP  project,  and  the  funding  for  it,  and  the  additional  ONR 
and  PMW-180  funding  approved  for  this  RTP,  will  allow  for  both  the  global  and  mesoscale  ensemble 
systems  to  be  more  fully  developed,  tested,  validated,  and  transitioned  to  operations  during  the  time 
period  of  FY07-FY09. 
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OBJECTIVES 


The  objective  of  this  project  is  to  incrementally  develop  and  test  an  integrated  multi-scale  ensemble 
modeling  framework  and  forecasting  system,  and  validate  and  transition  it  to  operations  at  the  Fleet 
Numerical  Meteorology  and  Oceanography  Center  (FNMOC).  This  system  will  be  a  state  of  the  art 
global  and  mesoscale  ensemble  forecasting  system  using  the  Navy  Operational  Global  Atmospheric 
Prediction  System  (NOGAPS)  and  the  Coupled/Ocean  Atmosphere  Mesoscale  Prediction  System 
(CO  AMPS®1).  It  will  incorporate  consideration  of  uncertainty  in  the  initial  state  (and  lateral 
boundaries  where  appropriate)  along  with  consideration  of  uncertainty  in  the  model  formulation.  It  will 
be  a  user-friendly,  easily  re-locatable  system  flexible  enough  to  incorporate  new  advances  in  data 
assimilation  and  post-processing  schemes.  The  system  will  provide  high  fidelity,  dynamically 
consistent  probabilistic  forecasts,  and  estimates  of  forecast  uncertainty  for  the  battlespace 
environment.  These  global  and  mesoscale  ensemble  forecasts  will  lead  to  superior  probabilistic 
weather  forecasts  that  will  directly  improve  tactical  decision  aids  (TDA)  for  support  of  FORCENET 
and  the  Sea  Power  21  pillars  (Sea  Basing,  Sea  Shield,  and  Sea  Strike),  and  for  applications  to 
homeland  security  and  the  global  war  on  terrorism  (GWOT). 

APPROACH 

Our  approach  is  to  take  existing  prototype  global  and  mesoscale  ensemble  forecast  systems,  built 
through  current  and  previous  NRL  in-house  funding,  and  incrementally  develop  these  individual 
systems  into  an  operationally  feasible,  next-generation,  multi-scale  ensemble  prediction  system.  This 
system  will  consist  of  (a)  the  representation  of  initial  and  model  uncertainty  in  NOGAPS,  (b)  the 
representation  of  initial,  lateral  boundary  condition  (LBC),  and  model  uncertainty  in  CO  AMPS,  (c)  an 
integrated,  robust  global-mesoscale  ensemble  infrastructure  (to  ensure  consistent  perturbations 
between  the  global  and  mesoscale  systems,  result  in  improvements  to  the  global  system  having  an 
immediate  impact  on  the  mesoscale  system,  and  promote  ease  of  use),  (d)  an  ensemble  verification 
system,  including  scorecard,  and  (e)  a  demonstration  of  performance  capabilities  in  an  operational 
setting.  The  system  will  be  thoroughly  tested  using  scientific  studies  and  norms  and  comprehensive 
ensemble  forecast  experiments. 

WORK  COMPLETED 

For  the  development  of  the  global  ensemble  system,  a  comparison  between  the  new  ET  (Bishop  and 
Toth  1999)  and  the  previously  operational  Bred  Vector  (BV,  Toth  and  Kalnay  1993)  scheme  has 
shown  the  ET  scheme  to  be  superior  under  many  different  metrics  (McLay  et  al.  2008).  The  ET 
scheme  was  transitioned  to  FNMOC  in  March  2008.  Research  is  now  underway  to  improve  the 
current  fit  of  the  initial  perturbations  to  the  analysis  error  variance  estimates  produced  by  the  data 
assimilation  scheme  using  an  alternative  basis  (McLay  and  Reynolds  2008).  Research  also 
investigated  the  impact  of  combining  the  ET  with  stochastic  convection  (Teixeira  and  Reynolds  2007; 
Reynolds  et  al.  2008)  including  the  impact  on  tropical  cyclone  track  forecasts.  Finally,  ongoing 
research  is  examining  how  varying  parameter  values  may  be  used  as  a  way  to  include  model 
uncertainty  in  ensemble  design. 

For  the  development  of  the  mesoscale  ensemble  system,  the  new  ET  method  for  generating  initial 
perturbations  has  also  been  implemented  within  CO  AMPS.  Perturbed  lateral  boundary  conditions  to 
drive  the  ensemble  were  obtained  from  the  global  NOGAPS  ET  ensemble  (McLay  et  al.,  2007a,  b). 

The  29-member  COAMPS  ensemble  system  has  been  tested  by  running  it  over  a  three  week  period  for 
summer  2005.  To  quantitatively  evaluate  the  performance  of  the  mesoscale  ensemble,  a  large  suite  of 
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diagnostics  was  developed  and  employed,  as  well  as  prototype  post-processing  software.  A  new, 
modified  set  of  ensemble  perturbations  was  developed,  tested,  and  compared  to  the  original  set  of 
perturbations.  The  testing  of  the  ability  of  the  multi-parameter  ET  ensemble  to  work  with  high 
resolution  nests  for  the  urban  meteorology  case  over  Tokyo  at  1.67  km  resolution  using  1 1  members 
was  extended  to  21  members  by  including  sea  surface  temperature  (SST)  perturbations.  The  initial 
technique  to  emulate  a  fully  coupled  air-ocean  system  was  to  use  off-line  COAMPS  simulations  to 
force  an  ocean  model,  and  then  force  the  atmospheric  ensemble  with  the  subsequent  SST  fields 
varying  every  hour. 

RESULTS 

While  the  first  version  of  the  global  ET  has  already  been  transitioned  to  operations,  there  are  still 
issues  with  the  current  ET  implementation  that  merit  further  research.  One  issue  is  that  the  ET  scheme 
has  too  little  ensemble  variance  in  the  tropics  when  compared  to  the  analysis  error  variance  estimate 
that  is  produced  by  the  operational  data  assimilation  system.  This  is  attributable,  in  part,  to  the  fact 
that  the  6-h  ensemble  forecast  perturbations  currently  serving  as  the  basis  for  the  ensemble  transform 
are  too  spatially  localized.  Recently,  two  alternative  bases,  one  composed  of  prior  72-h  ensemble 
forecasts  (EXP1)  and  one  composed  of  an  archive  of  randomly- sampled  ensemble  forecasts  (EXP2) 
have  been  compared  with  the  operational  basis  currently  being  used  (OPER).  Results  show  both 
experiments  give  improved  performance  under  a  variety  of  metrics,  including  increased  ensemble 
variance  in  the  tropics  and  improved  Brier  scores  (Fig.  1).  This  improved  performance  is  attributable 
to  the  reduced  localization  in  the  experimental  basis  perturbations  as  compared  to  the  operational 
perturbations  (McLay  and  Reynolds  2008). 


Fig.  1.  Results  for  the  operational  version  of  the  ET  (thick  solid  curves)  as  compared  with  two 
experimental  versions  using  alternative  bases  for  NOGAPS  forecasts  in  the  tropics.  The  left  panel 
shows  ensemble  variance  as  a  function  of forecast  time  in  terms  of  total  energy.  The  right  panel 
shows  Brier  score  (probabilistic  forecast  error  score)  as  a  function  of forecast  time  for  10-m  wind 

speed  in  the  tropics  exceeding  10  m  s'1. 

Another  way  to  improve  the  current  ET  implementation  is  through  inclusion  of  model  uncertainty. 
Experiments  combining  the  ET  with  stochastic  convection  show  improved  ensemble  performance 
under  a  variety  of  metrics  in  the  tropics,  with  little  impact  in  the  extra-tropics  (Reynolds  et  al.  2008). 
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It  is  shown  that,  because  the  ET  is  a  cycling  scheme,  direct  changes  to  the  forecasts  result  in  indirect 
changes  to  the  initial  perturbations  as  well.  Improvements  from  the  stochastic  convection  scheme  are 
attributable  to  both  direct  and  indirect  changes  at  short  forecast  times,  and  primarily  to  the  direct 
model  changes  at  longer  forecast  times.  The  utility  of  the  ET  stochastic  convection  ensemble  for 
tropical  cyclone  track  forecasting  has  also  been  examined.  It  is  found  that,  at  forecast  lead  times 
beyond  72  hours,  the  ET  with  stochastic  convection  provides  ensemble  mean  tropical  cyclone  track 
forecasts  that  are  comparable  in  skill  to  the  multi-model  consensus  and  official  track  forecasts  (Fig.  2). 


■  NOGAPS  High  Res. 

■  ET  STOC  ensemble  mean 
□  Multi-model  Consensus 
Q  Official  forecast 


24  48  72  96  120 

358  293  235  183  133  Number  of  Forecasts 


Fig.  2.  Average  forecast  error  (nm)  for  the  2005  Atlantic  Hurricane  season  as  a  function  of 
forecast  lead  time.  The  ET  Stochastic  ensemble  (red)  at  96  and  120  hours  is  comparable  in  skill  to 
the  multi-model  consensus  (green)  and  official  (pink)  forecasts. 


For  the  mesoscale  ensemble  system,  the  29-member  CO  AMPS  48-h  forecasts  for  the  45-km  CONUS 
domain  for  three  weeks  (25  June  to  17  July  2005)  has  been  used  to  develop  new  perturbed  physics 
parameters  and  new  post-processing  software.  Rank  histograms  (or  Talagrand  diagrams)  show  the 
rank  frequency  of  observations  relative  to  the  ensemble  forecasts,  and  are  a  useful  tool  to  evaluate  the 
reliability  of  probabilistic  forecasts.  Fig.  3  shows  the  Talagrand  diagram  for  the  29-member  COAMPS 
CONUS  case  for  850-hPa  u-wind  component,  v-wind  component,  and  air  temperature.  Thirty  ranks  are 
shown  because  the  unperturbed  control  member  is  included  in  the  analysis.  To  account  for  the  effect 
of  radiosonde  observation  error,  random  numbers  with  standard  deviation  equal  to  estimated 
observation  error  variance  (1.9  m  s'1  for  u-  and  v-  wind  components  and  1°C  for  air  temperature)  are 
added  to  each  of  the  forecasts  before  the  histogram  was  formed.  Forecasts  are  adjusted  by  adding  a 
bias  correction  constant  equal  to  the  21 -day  mean  of  the  observation  minus  the  forecast  (0.3  m  s'1  for  u 
and  v,  and  0.6°C  for  air  temperature). 
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Fig.  3.  Rank  frequency  of  observations  relative  to  the  ET  ensemble  (Talagrand  diagram)  for 
COAMPS  48-h  ensemble  forecasts  for  the  45-km  CONUS  domain  from  25  June  to  17  July  2005. 

Two  additional  experiments,  termed  version  1  (vl)  and  version  2  (v2)  were  conducted  in  which 
perturbations  to  the  COAMPS  planetary  boundary  layer  (PBL),  surface  layer,  microphysical,  and 
cumulus  physical  parameterizations  were  included  along  with  the  ET.  For  the  28  vl  perturbation 
members  of  the  ensemble,  the  perturbations  included:  3  PBL,  2  surface  flux,  2  microphysics,  13 
cumulus,  7  PBL/surface  flux/cumulus  combination,  and  1  PBL/surface  flux  combination.  For  this 
perturbation  set,  while  the  skill  of  the  ensemble  as  represented  by  the  spread-skill  relationship  was 
improved  with  perturbed  parameters  (as  compared  to  the  control),  some  members  exhibited  extreme 
errors.  This  is  illustrated  in  precipitation  error  metrics  shown  in  Fig.  4.  To  address  this  deficiency,  the 
initial  vl  perturbations  are  tuned  to  eliminate  members  with  unrealistically  large  statistics  and  new 
modified  members  for  physical  parameterization  perturbations  (v2)  are  developed.  The  v2  ensemble 
performance  is  generally  improved  over  vl  (Fig.  4  right  panels),  and  the  ensemble  mean  is  more 
skillful  than  the  control,  particularly  for  v2. 
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Fig.  4.  The  COAMPS  48-h  ensemble  precipitation  forecasts  for  the  4  5 -km  CONUS  domain  from  25 
June  to  17  July  2005 for  the  version  1  (vl)  perturbations  (left  panels)  and  version  2  (v2)  (right 
panels)  of  equitable  threat  score  (ETS)  (top),  percentage  of points  that  are  convective  (middle),  and 
bias  (mm)  (bottom).  The  numbers  in  the  plots  represent  member  number,  and  the  thick  black  line  is 

the  control  and  the  thick  red  line  is  the  ensemble  mean. 

Fig.  5  shows  the  spread-skill  relationship  for  precipitation  for  the  vl  and  v2  perturbation  ensembles. 
Note  that  the  vl  ensemble  attained  a  larger  spread  (over-dispersive),  but  the  relation  to  the  error 
variance  was  not  as  useful.  The  spread  associated  with  the  v2  ensemble  was  a  much  better  predictor  of 
error. 
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Fig.  5.  Ensemble  spread-forecast  error  relationship  for  precipitation  for  vl 

and  v2  perturbed  physics. 


IMPACT/APPLICATIONS 

Since  results  indicate  improved  performance  of  the  global  ET  system  over  the  previous  operational 
method  under  a  variety  of  metrics,  this  new  scheme  is  resulting  in  more  skillful  global  ensemble 
forecasts  for  the  Navy.  This  should  have  a  direct  impact  on  current  and  planned  ensemble-based 
products,  such  as  the  probability  of  gale-fore  winds,  as  well  as  other  products  that  are  currently  forced 
by  the  global  ensemble,  or  will  be  in  the  future,  including  the  mesoscale  atmospheric  ensembles,  and 
surface  ocean  wave  ensembles.  In  the  future,  it  is  hoped  that  these  more  skillful  probabilistic  tools  will 
lead  to  more  applications  of  probabilistic  forecasts  in  the  DoD,  including  ship  and  air  routing 
applications. 

TRANSITIONS 

Based  on  the  superior  performance  of  the  ET  as  compared  to  the  previously-operational  bred-vector 
scheme,  the  ET  scheme  was  transitioned  to  the  operational  global  ensemble  system  in  March  2008.  It 
is  anticipated  that  the  stochastic  physics  and  alternative  basis  improvements  will  be  transitioned 
through  6.4  to  operations  sometime  in  FY09.  In  addition,  as  part  of  the  Navy/ Air  Force  Joint 
Mesoscale  Ensemble  (JME)  ensemble  forecasting  project  we  have  aided  FNMOC  in  the  set  up,  and 
running  of  a  “beta-ops”  real  time  ensemble  forecasting  system  over  the  Korean  peninsula.  The 
CO  AMPS  ET  ensemble  has  been  running  continuously  on  the  HPC  DC3  system  since  late  June  2008. 
The  transition  to  operations  is  in  progress,  with  porting  to  the  FNMOC  OPAL  A2  system  underway. 
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RELATED  PROJECTS 


The  global  ET  and  Stochastic  Physics  research  began  in  preceding  years  under  the  NRL  base-funded 
project  6.1 -Quantifying  Limits  of  Atmospheric  Predictability  and  the  NOAA  funded  project  6.2 
Stochastic  Physics.  In  addition,  research  under  the  NRL  base-funded  project  6.2-Model  Discovery, 
which  uses  ensembles  to  understand  model  behavior  within  parameter  space,  will  lead  to  methods  to 
perturb  parameters  in  ensembles  and  be  transitioned  to  operations  through  this  work  unit  in  LY09. 

Lor  the  mesoscale  ensemble  development  related  projects  include  ONR  funded  6.2-Probabilistic 
Forecasting  of  High  Impact  Weather,  6.2-Hidden  Volatility  in  Environmental  State  Estimation,  and 
6.1 -Quantifying  Limits  of  Atmospheric  Predictability.  The  research  is  also  related  to  the  NOAA 
funded  projects  6.2  Stochastic  Physics  and  6.2  Huge  Ensemble  State  Estimation. 
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